Haptic sensation recording and playback

ABSTRACT

A system includes a video recorder configured to record video data, a sensor configured to sense movement of an object and output sensor data representative of the movement of the object, a transformer configured to transform the sensor data into a haptic output signal, a haptic output device configured to generate a haptic effect to a user based on the haptic output signal, a display configured to display a video, and a processor configured to synchronize the video data and the haptic output signal, and output the video data to the display and the haptic output signal to the haptic output device so that the haptic effect is synchronized with the video displayed on the display.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication Ser. No. 61/756,418, filed Jan. 24, 2013, the entire contentof which is incorporated herein by reference.

FIELD

The present invention is related to systems and methods for providinghaptic sensation recording and playback.

BACKGROUND

Electronic devices allow users to record videos that may capture boththe visual and audio aspects of an event. Electronic devices may also beprogrammed to provide haptic sensations while the user is watching avideo played on the electronic device. The haptic sensations arepreprogrammed so that when the video is played, the user may experiencehaptic effects in conjunction with the video to provide a more immersiveexperience for the user. Existing devices only allow the haptic effectsto be determined after the video has been recorded. Currently, creatinghaptic effects and sensations is a labor-intensive process that is notdone in real time.

SUMMARY

It is desirable to be able to record a video of an event while at thesame time record real-time aspects of the event so that the real-timeaspects of the event may be played back with the video as hapticsensations to provide an even more realistic and immersive experiencefor the user watching the video and feeling haptic sensation playback.Recording real-time data from an object in a scene, and reproducing itsexperience haptically, is a more intuitive way of “capturing” anexperience, when compared to offline artistic editing with programmingtools and writing synchronization code to attempt to synchronize thehaptic effects with the video. It is desirable to simplify the creationof such content.

According to an aspect of the present invention, there is provided asystem that includes a video recorder configured to record video data, asensor configured to sense movement of an object and output sensor datarepresentative of the movement of the object, a transformer configuredto transform the sensor data into a haptic output signal, a hapticoutput device configured to generate a haptic effect to a user based onthe haptic output signal, a display configured to display a video, and aprocessor configured to synchronize the video data and the haptic outputsignal, and output the video data to the display and the haptic outputsignal to the haptic output device so that the haptic effect issynchronized with the video displayed on the display.

In an embodiment, the system also includes an audio recorder configuredto record audio data, and a speaker configured to project sound. Theprocessor may be further configured to receive the audio data,synchronize the audio data with the video data and the haptic outputsignal, and output the audio data to the speaker so that the sound issynchronized with the video displayed on the display and the hapticeffect generated by the haptic output device.

In an embodiment, the video recorder, the audio recorder, and the sensorare part of the same electronic device. In an embodiment, the hapticoutput device, the display, and the speaker are part of the sameelectronic device. In an embodiment, the video recorder, audio recorder,the sensor, the haptic output device, the display, the speaker, and theprocessor are part of the same electronic device. In an embodiment, thehaptic output device and the display are part of the same electronicdevice.

In an embodiment, the processor comprises the transformer, and furthercomprises a decoder configured to synchronize the video data and thehaptic output signal.

In an embodiment, the sensor is selected from the group consisting of:an accelerometer, a gyroscope, and a contact pressure sensor.

In an embodiment, the system includes a plurality of sensors, and theplurality of sensors are selected from the group consisting of: anaccelerometer, a gyroscope, and a contact pressure sensor.

According to an aspect of the invention, there is provided a system thatincludes a display configured to display a video, a video recorderconfigured to record video data, a sensor configured to sense movementof an object and output sensor data representative of the movement ofthe object, a transformer configured to transform the sensor data into ahaptic output signal, and an electronic handheld device. The electronichandheld device includes a haptic output device configured to generate ahaptic effect to a user of the handheld device based on the hapticoutput signal, and a processor configured to synchronize the video dataand the haptic output signal, and output the video data to the displayand the haptic output signal to the haptic output device so that thehaptic effect is synchronized with the video displayed on the display.

According to an aspect of the present invention, there is provided amethod that includes recording a video with a video recorder, sensingmovement of an object being recorded in the video with a sensor,transforming the sensed movement into a haptic output signal, andsynchronizing the video and the haptic output signal.

In an embodiment, the method includes generating at least one hapticeffect based on the haptic output signal, and displaying the video on adisplay in synchronization with the haptic effect.

In an embodiment, the method includes recording audio with an audiorecorder and synchronizing the audio with the video and the hapticoutput signal.

In an embodiment, the method includes generating a haptic effect basedon the haptic output signal, displaying the video on a display, andprojecting the audio with a speaker so that the haptic effect issynchronized with the video displayed on the display and the audioprojected by the speaker.

BRIEF DESCRIPTION OF THE DRAWINGS

The components of the following Figures are illustrated to emphasize thegeneral principles of the present disclosure and are not necessarilydrawn to scale. Reference characters designating correspondingcomponents are repeated as necessary throughout the Figures for the sakeof consistency and clarity.

FIG. 1 illustrates an embodiment of a system for recording and providinga haptic effect;

FIG. 2 illustrates an embodiment of a wearable sensor that may be usedas part of the system of FIG. 1;

FIG. 3 illustrates a signal representative of vertical acceleration as afunction of time that may be measured by the sensor of FIG. 2;

FIG. 4 illustrates signals representative of acceleration as a functionof time that may be measured by the sensor of FIG. 2; and

FIG. 5 is a flow diagram of a method according to an embodiment of theinvention.

DETAILED DESCRIPTION

FIG. 1 illustrates a system 100 in accordance with an embodiment of theinvention. As illustrated, the system 100 includes one or more sensors102, which are configured to sense movement of an object and convert thesensed movement into sensor data, a video recorder 104 configured tocapture and record images, and an audio recorder 106 configured tocapture and record sound. In an embodiment, the sensor(s) 102, the videorecorder 104, and the audio recorder 106 may be part of the sameelectronic device. In an embodiment, the video recorder 104 and theaudio recorder 106 may be part of the same electronic device, and thesensor(s) 102 may be separate from the electronic device that includesthe video recorder 104 and the audio recorder 106. In an embodiment, thesensor(s) 102, the video recorder 104, and the audio recorder 106 may beseparate, stand-alone devices or part of separate, stand-alone devices.

A processor 110 is configured to process signals and data output by thesensor(s) 102, the video recorder 104, and the audio recorder 106, asdiscussed in further detail below. The system 100 also includes an inputtransformer 112, an output transformer 114, which may be part of theprocessor 110, and a decoder 116, which may also be part of theprocessor 110. Aspects of the input transformer 112, the outputtransformer 114, and the decoder 116 are discussed in further detailbelow.

As illustrated in FIG. 1, the system 100 also includes a haptic outputdevice 118 configured to output haptic effects to a user of the system,a display 120 configured to display images, such as the images capturedby the video recorder 104, and a speaker 122 configured to output sound,which may be the sound captured by the audio recorder 106. The hapticoutput device 118, the display 120, and the speaker 122 may be part ofan electronic playback device 130, as discussed in further detail below.In an embodiment, the haptic output device 118, the display 120, and thespeaker 122 may be separate devices that are configured to communicatewith each other through a wireless connection, for example. In anembodiment, the haptic output device 118 may be part of a wearabledevice, the display 120 may be part of a television, and the speaker 122may be a wireless speaker that is separate from the display 120.

The haptic output device 118 may include an actuator, for example, anelectromagnetic actuator such as an Eccentric Rotating Mass (“ERM”) inwhich an eccentric mass is moved by a motor, a Linear Resonant Actuator(“LRA”) in which a mass attached to a spring is driven back and forth,or a “smart material” such as piezoelectric, electro-active polymers orshape memory alloys, a macro-composite fiber, actuator, anelectro-static actuator, an electro-tactile actuator, and/or anothertype of actuator that provides a physical feedback such as a haptic(e.g., vibrotactile) feedback. The haptic output device 118 may includenon-mechanical or non-vibratory devices such as those that useelectrostatic friction (ESF), ultrasonic surface friction (USF), orthose that induce acoustic radiation pressure with an ultrasonic haptictransducer, or those that use a haptic substrate and a flexible ordeformable surface, or those that provide projected haptic output suchas a puff of air using an air jet, and so on.

Electronic memory 124 may be used to store data sensed by the sensor(s)102, electronic memory 126 may be used to store data that is recorded bythe video recorder 104, and electronic memory 128 may be used to storedata that is recorded by the audio recorder 106. The memory 124, 126,128 may include one or more internally fixed storage units, removablestorage units, and/or remotely accessible storage units. The variousstorage units may include any combination of volatile memory andnon-volatile memory. The storage units may be configured to store anycombination of information, data, instructions, software code, etc. Inembodiments in which the sensor(s) 102, the video recorder 104, and theaudio recorder 106 are part of the same electronic device, the memory124, 126, 128 may be co-located. In embodiments in which the videorecorder 104 and the audio recorder 106 are part of the same electronicdevice, the memory 126, 128 may be co-located.

In an embodiment, a user may record video and/or audio of a scene orevent using the video recorder 104 and/or the audio recorder 106. In anembodiment, the video recorder 104 and the audio recorder 106 may bepart of the same recording device, such as a video camcorder, a smartphone, etc. The video and audio that is recorded may be stored in theelectronic memory 126, as discussed above. In an embodiment, thesensor(s) 102 may be placed on an object of interest, such as an objectbeing recorded in the video.

As discussed above, the data generated by the sensor(s) 102 may bestored in the electronic memory 124. In addition, the data generated bythe sensor(s) 102 may be transformed by the input transformer 112 priorto being stored in the electronic memory 124, as illustrated in FIG. 1.The transformation of the sensor data is considered to be an optionalstep and whether the transformation is needed may depend on the natureof the sensors being used. Details of an embodiment of the sensor 102are discussed in further detail below.

The decoder 116, which may be part of a media player configured toplayback the video, i.e. media file, is configured to read the datagenerated by the sensor(s) 102 from the electronic memory 124, andassociate the data temporally with the audio data and video data thatwere recorded and stored in the electronic memory 126. During mediaplayback, the decoder 116 may pass the sensor data through an outputtransformer 114 configured to transform the sensor data into a hapticoutput signal or haptic sensory commands, which include but are notlimited to, vibration, surface friction modulation, skin pinch, skinsqueeze, etc. The decoder 116 may be configured to synchronize thehaptic output signal that was transformed from the sensor data with thevideo data and the audio data so that the haptic effect is synchronizedwith the video and audio during playback. In an embodiment, thesynchronization may be completed by ensuring that time is the same inthe video data, the audio data, and the haptic effect during playback.

The processor 110 may be a general-purpose or specific-purpose processoror microcontroller for managing or controlling the operations andfunctions of the system 100. For example, the processor 110 may bespecifically designed as an application-specific integrated circuit(“ASIC”) to control output signals to the haptic output device 118 toprovide haptic effects. The processor 110 may be configured to decide,based on predefined factors, what haptic effects are to be generated,the order in which the haptic effects are generated, and the magnitude,frequency, duration, and/or other parameters of the haptic effects. Theprocessor 110 may also be configured to provide streaming commands thatmay be used to drive the haptic output device 118 for providing aparticular haptic effect. In some embodiments, the processor 110 mayactually include a plurality of processors, each configured to performcertain functions within the system 100. The processor 110 may alsoinclude memory that includes one or more storage devices that mayinclude haptic effect profiles, instructions for how the haptic outputdevice 118 is to be driven, and/or other information for generatinghaptic effects. In an embodiment in which the entire system 100illustrated in FIG. 1 is part of a single electronic device, the memory124, 126 may be part of the processor 110.

The haptic output signal may then be transmitted from the processor 110,e.g., from the decoder 116 of the processor 110, to the haptic outputdevice 118 so that the person(s) experiencing the media through theelectronic playback device 130 that includes the haptic output device118 may more fully experience the event being played back. Theelectronic playback device 130 may be any device, such as an electronichandheld device, such as a mobile phone, gaming device, personal digitalassistant (“PDA”), portable e-mail device, portable Internet accessdevice, calculator, tablet, etc. The electronic playback device 130 mayinclude, but is not limited to, a handheld device with the display 120,which may be a high definition display, that displays the media, ahandheld object that is capable of producing haptic sensations oreffects, or an object attached to the user's body, leaning up to theuser's body, or otherwise able to transmit tactile sensations and hapticeffects to the user.

In an embodiment, the processor 110 and the haptic output device 118 maybe part of an electronic handheld device, which may be a phone or atablet, and the electronic handheld device is configured to output thevideo data to a separate display 120, which may be a television.

In an embodiment, the system 100 may include a mobile phone having agyroscope, a compass, and three-axis accelerometer sensors for thesensors 102, as well as a built-in camera for the video recorder 104. Inthis instance, all of the components illustrated in FIG. 1, includingthe data recording sensors 102, video recorder 104, audio recorder 106,processor 110 including the decoder 116 and output transformer 114,haptic output device 118, display 120, speaker 122, input transformer112, and electronic memory 124, 126, may be self-contained, and theentire system 100 may be affixed to the person or a piece of equipmentperforming the activity of interest.

In an embodiment, a first-person perspective video camera may be mountedto a helmet or piece of equipment performing the activity of interest,and the video camera may incorporate a number of data sensors 102, suchas accelerometers, a global positioning system (“GPS”), and gyroscopes,the input transformer 112, if needed, the electronic memory 124, thevideo recorder 104, the audio recorder 106, and the electronic memory126. The remaining parts of the system 100, such as the processor 110including the decoder 116 and output transformer 114, the haptic outputdevice 118, the display 120, and the speaker 122, may be located in aseparate playback device, such as the electronic playback device 130, asdiscussed above.

In an embodiment, the sensor(s) 102, which may include one or more datasensors, such as accelerometers, GPS, etc., may be affixed either to theperson or to equipment performing the activity of interest. Thesensor(s) 102 may be contained in a sensor box, or some other containerthat is configured to protect the sensor(s) 102. The sensor box may havedata recording means, such as the input transformer 112 and theelectronic memory 124, built-in, or may rely on a data connection tosecondary device (such as a mobile device) to record the data during theactivity.

In an embodiment, the sensor(s) 102 may be installed on a person who isthe subject of the video, and the video recorder 104, and optionally theaudio recorder 106, may be operated by another person. For example, theperson who is the subject of the video may be a snowboarder and at leastone sensor 102 may be attached to the snowboarder's boot and/or on otherpieces of clothing or equipment, such as the snowboarder's snowboard.The sensor(s) 102 may include accelerometers configured to provide theg-force accelerations experienced by the snowboard, gyroscope sensorsconfigured to provide the orientation of the snowboard, and contactpressure sensors configured to provide the load applied to the snowboardby the snowboarder's boots. In an embodiment, the audio recorder 106 maybe attached to an article of clothing or equipment, such as a helmet ofthe snowboarder so that the sound of the snowboard passing over the snowmay be captured.

In an embodiment, a recording session may be initiated in which thestream of acceleration data, gyroscope data, and contact pressure datais recorded alongside the video and audio data. The video recorder 104and/or audio recorder 106 may be mounted to the snowboarder's helmet. Inan embodiment, the video recorder 104 and/or audio recorder 106 may bean array of video cameras mounted at various locations in a snow park'shalf pipe, where the snowboarding activity is to be performed. In anembodiment, the video recorder 104 may be an array of video cameraslocated at various locations in a snow park's half pipe and the audiorecorder 106 may be mounted to an article of clothing or equipment ofthe snowboarder, as described above. Other combinations of the locationsof the sensor(s) 102, video recorder 104, and the audio recorder 106 maybe used and may depend on the activity being recorded so that an optimumhaptic/audio/visual experience may be realized by the user of theelectronic playback device 130 during playback. The synchronization ofall of these data streams may be managed by recording software, whichmay reside in the processor 110 of the system 100 illustrated in FIG. 1.

In an embodiment, flexible container formats, such as MPEG-4, that allowfor the storage of data other than video and audio in a single filecontainer, may be used. In such an embodiment, a particular set ofencoders may be needed to place the sensor data into the MPEG-4 fileduring recording. In an embodiment, special software may be written tostore the non-audio and video (A/V) sensor data in a separate file, butwith special markers in the sensor data to allow for propersynchronization at playback time. In this embodiment, very little inputtransformation may need to be applied, beyond shaping the sensor data toconform to the limitations of the designed recording format. The exactformat may be determined by the implementer. Once the snowboarder hascompleted his or her activity, the recording may be stopped. The MPEG-4file may be closed, and all of the sensor data may reside in the MPEG-4file.

In an embodiment, the playback device may be the electronic playbackdevice 130 of FIG. 1, and may be in the form of a mobile phone or tablethaving the display 120, the speaker 122, and a vibration device as thehaptic output device 118 to provide the haptic effect. In an embodiment,the playback device may be a gaming console connected to a televisionhaving the display 120 and the speaker 122, and also connected to agaming peripheral, such as a gamepad, that includes the haptic outputdevice 118 to provide the haptic effect.

Either at a later time, or concurrently with the activity beingperformed, one or more viewers may be interested in experiencing theactivity. To playback the activity, the viewer may launch theappropriate playback software on their playback device with theobjective of experiencing the performer's activity. In an embodiment,the playback software may include a player software application thatincorporates the sensor decoding scheme performed by the decoder 116, aswell as output transform software that may be run by the outputtransformer 114, in order to transform the sensor data into a hapticoutput signal suitable for the haptic output device 118 in the playbackdevice. In an embodiment, a player software application may incorporatethe sensor decoding scheme. The player software may rely on the outputtransform software being resident or otherwise pre-installed on theplayback device, and such output transform software may transform thesensor data into the haptic output signal suitable for the haptic outputdevice 118 in the playback device. In other words, the outputtransformer 114 and/or decoder 116 may be located on the playbackdevice.

In an embodiment, a player software application may rely on the playbackdevice's operating system software to perform the media playback, whichincorporates the sensor decoding scheme. The operating system softwaremay rely on the output transform software being resident or otherwisepre-installed on the playback device, such output transform softwaretransforming the sensor data into a haptic output signal suitable forthe haptic output device 118 in the playback device. The viewer may thenexperience haptic sensations associated with the viewing of theperformance, such haptic sensations being produced by the outputtransform software.

In an embodiment, the output transform software may include some of thesensor data, while other sensor data is ignored. For example, thegyroscope data may be included, while the contact pressure sensor datamay be ignored. The magnitudes of the acceleration data may be computedby computing the vector magnitude of X-Y-Z acceleration components. Thisacceleration magnitude signal may then be band-pass filtered such thatonly acceleration magnitude signals in the 20-200 Hz range are output,with other frequency content filtered out. The filtered accelerationsignal may then be put through a soft-knee compression algorithm togently clip the output magnitudes between +6 and −6 gees, then allnegative signals may be zeroed/ignored, and the resultant compressed,single-sided output signal may be used to control a magnitude ofvibration provided by the haptic output device between 0-100% vibration.In this way, the viewer may feel a representation of the accelerationvariants felt by the performer's snowboard.

In an embodiment, the acceleration data, gyroscope data, and contactpressure sensor data may be combined into a haptic output signal in thefollowing way. Whenever the contact pressure is low, the snowboarder maybe assumed to be jumping and therefore all haptic output is zeroed out.When the contact pressure is significant, the contact pressure may bemultiplied by the acceleration magnitude. This output may then bemultiplied by the change in orientation signal (the first timederivative of the orientation heading, derived from the gyroscopesignal) to obtain an “activity intensity” signal. This signal may thenbe filtered and compressed in such a way so as to obtain a reasonablerepresentation of the range of sensations experienced by the performerduring the snowboarding activity, and the signal may be applied to theplayback device's haptic output device 118.

In an embodiment, whenever the contact pressure is low, the snowboardermay be assumed to be jumping and therefore all haptic output is zeroedout. When the contact pressure is significant, the accelerationmagnitude may be band pass filtered in the 100-200 Hz range, forexample, and its output range may be highly compressed using a hard-kneecompressor. This signal may be applied to the playback device's hapticoutput device 118.

The viewer holding or otherwise in contact with the haptic output device118 may then feel the sensations produced by the output transformer 114and output by the haptic output device 118.

Example

Adequate haptic effects/sensations were generated using video andsensors recordings that were acquired at the same time. A three-axisaccelerometer was used for the sensor 102 and was fixed to a firstsnowboarder's boot using a strap-on, as illustrated in FIG. 2. Thethree-axis accelerometer was oriented so that the X axis was generallyaligned from up to down, the Y axis was generally aligned from back tofront, and the Z axis was generally aligned from right to left, withrespect to the orientation of the snowboarder's boot, as illustrated inFIG. 2. A video camcorder was used for the video recorder 104 and theaudio recorder 106 and was operated by a second snowboarder, who wasfollowing the first snowboarder in order to generate the video and audiodata as the first snowboarder generated the sensor data.

After all of the data was recorded, the registered data from the sensor102 (accelerometer) was synchronized with the video images and audiodata recorded by the video recorder 104 and the audio recorder 106. Thefirst snowboarder was asked to accomplish four jumps before starting hisdescent. The timing of the four jumps was detected by analyzing thevideo feed from the video recorder 104 image by image. In addition, thetiming of the jumps were detected by analyzing the vertical accelerationsignal A_(x) from the sensor 102, as the jumps translated into fourconsecutive peaks, as illustrated in FIG. 3. Because the accelerometerreadings were time-stamped, the accelerations could be extracted andsynchronized with the entire video.

After synchronizing the data, specific events called “bumps” that wererelated to the moment of landing after a jump in the air or whilehopping on any object (e.g. rails) were automatically detected. Thesebumps may be characterized by creating a sudden peak in the accelerationsignal related to the direction of the landing. In snowboarding, alanding may be vertical or ‘inclined’, so the magnitude of the vectorA_(x)+A_(y) for each acceleration sample was calculated. Whenever themagnitude value was above a certain threshold, the sample was consideredto represent a bump. For each bump detected, a bump effect wasintroduced in the haptic track as a haptic effect at the moment of itsoccurrence. The effect strength was relative to the bump magnitude.

The texture of the surface on which the snowboarder was sliding wascaptured and represented in the haptic track using the followingapproach. Each of the three acceleration signals A_(x), A_(y), andA_(z), represented by 401, 402, 403 in FIG. 4, respectively, wastranslated from the time-domain into the frequency domain using a FastFourier transform (“FFT”). The three frequency domain signals were thenadded as vectors to form a single signal A_(transf). This latter signalwas finally translated back to the time domain into the A_(trans)signal, as represented by 404 in FIG. 4. This technique of transformingthree signals into one signal is known in the art, and thereforespecific details of the transformation are not provided herein.

The single A_(trans) signal 404 was then up-sampled from its originalsampling rate (i.e. 400 Hz) to 8000 Hz by linear interpolation. In thisinterpolated signal, all of the values corresponding to bumps were fixedto 0. The resulting signal was then multiplied by sine waves with randomfrequencies between 120 Hz and 270 Hz, and then normalized and copieddirectly in the haptic track that would thus contain the texture andbump effects. The haptic track could then be played back in the playbackdevice by the haptic output device 118. In an embodiment, when theacceleration signal is under sampled (e.g., at a sampling rate <100 Hz),the up-sampling of the transformed signal to 8000 Hz may be done byadding a white Bayesian noise between the captured data samples insteadof interpolation.

In an embodiment, the “swings” of the snowboarder, i.e. left to rightdirection changes or “s-curves”, may be detected and a relevant hapticeffect may be created and added to the texture and bumps. In anembodiment, accelerations of the snowboard itself and not thesnowboarder may be used. In an embodiment, the acceleration signals maybe shifted in the frequency domain by 150 Hz before being added into theA_(transf) signal.

FIG. 5 illustrates a method 500 in accordance with embodiments of theinvention described herein. The method 500 starts at 502. At 504, videois recorded with a video recorder, such as the video recorder 104described above, and movement of an object being recorded by the videorecorder is sensed with a sensor, such as the sensor 102 describedabove. At 506, the movement sensed by the sensor is transformed into ahaptic output signal, and at 508, the video and the haptic output signalare synchronized. The method may end at 512. In an embodiment, themethod may further include generating a haptic effect based on thehaptic output signal with, for example, the haptic output device 118described above, and displaying the video on a display, such as thedisplay 120 described above, so that the haptic effect is synchronizedwith the video displayed on the display during playback on a playbackdevice, such as the electronic playback device 130 described above. Inan embodiment, audio may also be recorded, synchronized, and projectedduring playback on the playback device.

Embodiments of the invention described herein allow content creators torecord media, such as video and/or audio, and other data, potentiallytransform it, and transmit haptic sensations associated with the mediaas experienced and recorded by the content creator. Recording andtransforming real-world sensor data to produce haptic output is a morecost-efficient way to generate haptic content (when compared tohand-authored synthesis of haptic signals), and may producecontextually-appropriate “realistic” sensations, with the proper outputtransform.

The embodiments described herein represent a number of possibleimplementations and examples and are not intended to necessarily limitthe present disclosure to any specific embodiments. For example, in theabove-described embodiments, the common capability is one of collectingand recording sensor data beyond the typical video and audio feed.Various modifications can be made to these embodiments as would beunderstood by one of ordinary skill in the art. Any such modificationsare intended to be included within the spirit and scope of the presentdisclosure and protected by the following claims.

What is claimed is:
 1. A system comprising: a video recorder configuredto record video data; a sensor configured to sense movement of an objectand output sensor data representative of the movement of the object; atransformer configured to transform the sensor data into a haptic outputsignal; a haptic output device configured to generate a haptic effect toa user based on the haptic output signal; a display configured todisplay a video; and a processor configured to synchronize the videodata and the haptic output signal, and output the video data to thedisplay and the haptic output signal to the haptic output device so thatthe haptic effect is synchronized with the video displayed on thedisplay.
 2. The system according to claim 1, further comprising an audiorecorder configured to record audio data, and a speaker configured toproject sound, wherein the processor is further configured to receivethe audio data, synchronize the audio data with the video data and thehaptic output signal, and output the audio data to the speaker so thatthe sound is synchronized with the video displayed on the display andthe haptic effect generated by the haptic output device.
 3. The systemaccording to claim 2, wherein the video recorder, the audio recorder,and the sensor are part of the same electronic device.
 4. The systemaccording to claim 2, wherein the haptic output device, the display, andthe speaker are part of the same electronic device.
 5. The systemaccording to claim 2, wherein the video recorder, audio recorder, thesensor, the haptic output device, the display, the speaker, and theprocessor are part of the same electronic device.
 6. The systemaccording to claim 1, wherein the haptic output device and the displayare part of the same electronic device.
 7. The system according to claim1, wherein the processor comprises the transformer, and furthercomprises a decoder configured to synchronize the video data and thehaptic output signal.
 8. The system according to claim 1, wherein thesensor is selected from the group consisting of: an accelerometer, agyroscope, and a contact pressure sensor.
 9. The system according toclaim 1, wherein the system comprises a plurality of sensors, andwherein the plurality of sensors are selected from the group consistingof: an accelerometer, a gyroscope, and a contact pressure sensor.
 10. Asystem comprising: a display configured to display a video; a videorecorder configured to record video data; a sensor configured to sensemovement of an object and output sensor data representative of themovement of the object; a transformer configured to transform the sensordata into a haptic output signal; and an electronic handheld devicecomprising a haptic output device configured to generate a haptic effectto a user of the handheld device based on the haptic output signal; anda processor configured to synchronize the video data and the hapticoutput signal, and output the video data to the display and the hapticoutput signal to the haptic output device so that the haptic effect issynchronized with the video displayed on the display.
 11. The systemaccording to claim 10, further comprising an audio recorder configuredto record audio data, and a speaker configured to project sound, whereinthe processor is further configured to receive the audio data,synchronize the audio data with the video data and the haptic outputsignal, and output the audio data to the speaker so that the sound issynchronized with the video displayed on the display and the hapticeffect generated by the haptic output device.
 12. A method comprising:recording a video with a video recorder; sensing movement of an objectbeing recorded in the video with a sensor; transforming the sensedmovement into a haptic output signal; and synchronizing the video andthe haptic output signal.
 13. The method according to claim 12, furthercomprising: generating at least one haptic effect based on the hapticoutput signal; and displaying the video on a display in synchronizationwith the haptic effect.
 14. The method according to claim 12, furthercomprising recording audio with an audio recorder and synchronizing theaudio with the video and the haptic output signal.
 15. The methodaccording to claim 14, further comprising generating a haptic effectbased on the haptic output signal, displaying the video on a display,and projecting the audio with a speaker so that the haptic effect issynchronized with the video displayed on the display and the audioprojected by the speaker.
 16. The method according to claim 12, whereinthe sensor is selected from the group consisting of: an accelerometer, agyroscope, and a contact pressure sensor.